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ABSTBACT 

TO provide diagnostic feedback for college faculty, a 
faculty evaluation rating scale was developed and used by college 
students. The instrument consisted of 2 global items— rate the 
instructor, rate the course: 5 questions on general teaching 
characteristic-- presentation, enthusiasm, discussion, organization, 
and personal attention: and 4 2 low inference items. The latter were 
specific, operational descriptions of teaching techniques. The five 
aeneral characteristics correlated ,30 or higher with the "rate the 
instructor" item? they frequently correlated at that level with the 
"rate the course" item. The low-inference items having the highest 
correlation! with the general characteristics are listed. The highly 
significant, partial correlations found between enthusiasm-discussion 
and presentation -organization suagest that there are not five but 
only two unique general characteristics: furthermore, both paris of 
characteristics have some low-inference behaviors in common. Bating 
forms need consist only of overall evaluation questions, for 
administrative purposes, and one question for each of two or three 
aeneral characteristics. From the results of the general 
characteristics ratings, the instructor receives a list of correlated 
specific items as feedback. (CP) 
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student Ratings of College Instruction; Some Low Inference Variables 
ROBERT J. STEVENS, University of Illinois 
BARAK V. ROSENSHINE, University of Illinois 

The objective of tliis study was to begin to develop a set of spe- 
cific, low inference items which are related to general characteristic 
factors commonly used on college instructor rating forms. The results 
delineate some specific items related to factors of presentation, organiza- 
tion, discussion, enthusiasm and personal attention. These low Infer- 
ence items can be used to provide diagnostic feedback to the instruc- 
tors concerning their teaching performance, and this feedback could 
provide the instructors with some specific information on how to improve 
their performance. 
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Student Ratings of College Instruction: Some Low Inference Variables 

ROBERT J. STEVENS 
BARAK V. ROSENSHINE 
University of Illinois 

Student ratings of instruction are used to evaluate and improve 
college Instruction (Smock and Crooks, 1973}. The purpose of this 
study was to begin to develop a set of specific, low inference items 
which are correlated with the general characteristic factors commonly 
used on college Instructor rating forms. The low inference items can 
then be used to provide diagnostic feedback to instructors concerning 
their teaching performance. 

Student ratings have consistently been shown to be a reliable 
means of evaluation (Costin, Creenough and Menges, 1971; Doyle, 1975; 
and Feldman, 1978). They have also been shown to be valid. Both 
overall ratings and particular factor ratings have been positively and 
significantly correlated with achievement, as measured by content- 
specific criterion tests (Braskamp, Caulley, and Costin, 1979; Bryson, 
1975, Centra, 1977; Frey, 1973, 1976; Gessner, 1973; Marsh, 1977; 
Marsh, Fleiner and Thomas, 1975; and McKeachie, Un and Mann, 1971). 

However, ratings on factor scores are too vague to provide the 
instructor with specific, usable feedback. When given feedback on 
these high inference items, instructors may find it difficult to translate 
the results into behaviors. Many factor analysis studies have investi- 
gated items which are clustered with various factors, but in many cases 
those items also lack the specificity necessary for providing useful 
feedback. 
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Smock and Crooks (1973) categorized items into Level 1, Level 2, 
and Level 3, as levels of increasing specificity. Level 3 was described 
as specific items much like Rosenshine's (1970) low inference. It Is 
these low inference items which have the specificity required to give 
the Instructor useful feedback which can be easily translated into 
behaviors. 

In an investigation of these levels of specificity, Brandenburg, 
Derry and Hengstler (Note 1) used a hierarchical factor analysis on the 
results from student ratings. They found three levels of specificity, 
much like those presented by Smock and Crooks. 

a) Global - general, summative evaluation. 

b) General characteristic - general areas or attributes of instruc- 
tion. 

c) Specific - specific attributes or aspects of teaching. 
(Brandenburg, et. al.. Note 1, p. 4) 

To facilitate readability and ease understanding, these categories will be 
used throughout the remainder of this paper. 

The amount of effect that student ratings can have on teacher 
effectiveness or teacher behavior has not been fully investigated. 
There is some evidence that teachers who are presented with accurate 
and specific feedback can use this information to change their behavior 
(Good and Brophy, 1973; Moore and Schaut, (Note 3), and Pambooklan, 
1976). Experimental studies have been done by Centra (Note 2) and 
McKeachie (1975) using student ratings as the source of specific feed- 
back to college instructors. In both studies the instructors who re- 
ceived specific feedback showed significlant Increases in the direction 
Indicated by the feedback, as measured by higher final student ratings, 
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thus suggesting that specific, usable teedback can help instructors 
Improve their teaching. 

Method 

In order to develop a set of specific, low inference Items which 
could be used as diagnostic feedback to instructors, the authors devel- 
oped a rating form containing both global, general characteristics and 
specific items. The general characteristics chosen were those which 
were most frequently correlated with achievement In previous research. 
In attempting to write the new specific Items, it was desired that they 
not only be low inference Items, but that they also provide the instruc- 
tor with some information on which s/he could act. (In writing each 
Item we asked ''After reading ..lis item, would the Instructor know what 
to do to change his/her behavior?") 

The rating instrument consisted of two global questions (rate the 
instructor, rate the course) , five questions rating general characteris- 
tics (presentation, enthusiasm, discussion, organization and personal 
attention), and 42 specific Items. Due to the length of the instrument, 
it was split into two forms with each form containing the global and 
general characteristic Items, but only half of the specific items, the 
second half of which were on the other form. The forms used a five 
point response scheme ("almost always occurred" to "almost never 
occurred") for the general characteristic Items and the specific items. 
The global items used a five point response scheme of "excellent" to 
"poor" . 

The questionnaires were then given to students In botany, educa- 
tional psychology and music classes at the University of Illinois during 
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the fall semester 1976 and the spring semester 1977. Each student was 
randomly assigned to either form A or form B and asked to complete the 
questionnaire evaluating the instructor of the course. The students 
were told that their responses would remain anonymous, and that they 
were not to identify themselves on the answer sheets. The sample sizes 
for the two forms were 136 and 119 students respectively. 

Results 

There is some question as to which unit of analysis is most appro- 
priate to use when analyzing student ratings. Although the student is 
typically the unit of analysis, it has been argued that using the class- 
room would be more appropriate. "Since the focus of the ratings Is the 
instructor, it might be argued that the classroom is a more appropriate 
unit of analysis. Accordingly, classroom Item means would be the basic 
data" (Linn, Centra and Tucker, 1975, p. 278). However, a study 
designed to compare the factors resulting from total group, between 
group and within group analyses, by Linn, et al . (1975), did not 
support this argument. Instead the researchers found that the "total 
group fector solution provided a very good fit to both the between and 
within covariance matrices" (Linn, et al ., p. 288) and thus the "factors 
from previous total group analyses would be expected to provide good 
approximations to the between group covariances" (Linn, et al ., p. 
288). Thus the use of the student as the unit of analysis provides a 
good approximation of the results of analysis by class, and therefore is 
used as the unit of analysis In this study. 

Initially the results showed a strong correlation between the global 
ratings and the general characteristic items, suggesting evidence of a 
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halo effect. The five general characteristics all correlated .30 or better 
with the "rate the instructor" item on both forms, as well as frequently 
correlating at that level with the "rate the course" item. 



Insert Table 1 Here 



In order to eliminate this halo effect upon the students' evaluations o^ 
the general characteristic and specific items, a partial correlation was 
performed, partialling out the two overall evauation items from the 
correlations of the general characteristic and specific items. 

The .01 level of signficance was used as a more stringent criteria 
for significance to help to eliminate the overlap across the general 
concepts. This made the results more clear and interpretable, how- 
ever, some overlap did remain as will be discussed later. The list of 
the specific items signflclantly correlated with particular general charac- 
teristic items (as presented in Table 2) suggest some of the behaviors 
related to ratings on that general characteristic item. 



Insert Table 2 Here 



Using these results, and grouping the specific Items with those 
variables with which they had the highest correlation (thus having only 
one entry per specific item) provides a set of operational ized specific 
behaviors under each general characteristic (presented below). 
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Presentation: 



The Instructor pointed out what was Important to learn in 
each class session. 

The Instructor summarized the material presented in each 
class. 

The instructor defines students' responsibilities in the 
course. 

The instructor used periodic reviews when making logical 
transitions. 



Personal Attention: 

Concern was shown for Individual differences. 

The Instructor provided appropriate material for differing 
rates of progress. 

The instructor checked frequently on students' understanding 
of the material. 

Enthusiasm: 

The instructor was a dynamic and energetic peison. 

The instructor praised the work of the students. 

The Instructor spoke in a monotone, rarely showing expres- 
sion in his voice, (negative) 

The Instructor is clear and concise in presentation and 
explanation of the material. 

The readings were relevant to the course objectives. 
Discussion: 

The Instructor used different methods and materials. 

The Instructor used student responses/contributions in de- 
veloping the lesson. 

The Instructor made an effort to show the interesting nature 
of the topic. 
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Thfc instructor used a variety of teaching methods. 
The instructor provided alternative ways of learning the 
course material. 

The material was too superficial to adequately develop my skill 
on concepts, (negative) 

The instructor used gestures while teaching 

thus the teachers who were rated high on the general characteristic 
Items more frequently exhibited those behaviors listed under that 
general characteristic. For example, an Instructor who was rated high 
on presentation was one who pointed out what was Important to learn, 
summarized the material presented, defined students* responsibilities 
and used periodic reviews when making transitions. 

The fact that some of these general characteristics also show 
significant Intercorrelatlons with each other, even after partialling out 
the overall evaluations, is also of Interest. These results, presented in 
Table 3, suggest that there Is not a clear differentiation between these 
general concepts. 



Insert Table 3 Here 
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The highly significant partial correlations (p<.01) foudn between the 
variables of enthusiasm and discussion, and the variables of present- 
ation and organization, found on both forms provide evidence that there 
is a great deal of overlap bet een these general characteristics. Like- 
wise there is evidence that the characteristic of personal attention has 
some overlap with the other genreal characteristics. In particular the 
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presentation and enthusiasm variables. (However, these results for 
personal attention were not consistent across the two forms.) This may 
then be evidence that there are not five, but rather only two unique 
factors involved in such ratings. (One study, done by Brandenburg 
and his associates (Note 1) supports this point of view.) These results 
suggest factors of presentation/organization, and enthusiasm/discussion, 
with personal attention not clearly falling in with either one. This 
result Is given further support by the fact that these pairs of variables 
also have some low inference behaviors in common (See Table 2). 
These concurrent specific behaviors for the presentation/ organization 
concepts are: 

The instructor spent time in material relevant to course objectives. 

The Instructor was able to answer questions clearly and concisely. 

The instrrctor pointed out what was important to learn. 

The instructor defines students' responsibilities. 

The instructor developed eye contact with the students. 

Similarly for the enthusiasm/discussion concepts the common Items were: 

The instructor was a dynamic and energetic person. 

The instrucff»r used student responses/contributions in developing 
the lesson. 

And as suggested by the Intercorrelations, the personal attention con- 
cept had items which were common to enthusiasm: 

The Instructor was a dynamic and energetic person, 
and common to presentation: 

The Instructor spent time In material relevant to course objectives. 

The Instructor was adequately prepared for each class. 

These results are by no means conclusive on this point, and 
further study is needed to determine how many factors are involved In 
these ratings. 



Discussion 

The results of this study provide a list of specific (low inference) 
behaviors which are related to students' evaluations on particular char- 
acteristics. The specifics can provide the instructor with potentially 
valuable feedback concerni.tg these student ratings. By providing the 
instructor with the behaviors related to the general characteristics 
along with the raw scores (or percentile scores) , he will have specific 
Information on which to act. This information can provide some help to 
the Instructor attempting to change his classroom behavior. An in- 
structor who is rated low on a general characteristic item can refer to 
the specific behavior related to the general characteristic. The in- 
structor can then use this feedback in such a way as to change his 
behavior In an attempt to improve in the area of that particular general 
characteristic. For example, if the instructor was rated low on the 
presentation characteristic, he should attempt to exhibit more frequently 
the behaviors related to that variable. Therefore the instructor should 
point out what is important to learn in class; summarize the material 
presented in class; define students' responsibilities; and so forth. 
Although this study does not totally exhaust the set of behaviors which 
may be related to these variables, it does provide an Initial step in 
delineating the low inference behaviors related to these high Inference 
variables. 

Another useful result of this study, and studies like It, is that it 
suggests that rating forms need consist only of overall evaluation ques- 
tions (for administrative purposes) and one question for each of two or 
three general characteristic items. From the results of the general 
characteristic ratings the instructor can be provided with the list of 
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related specifics as prescriptive feedback. The addition of specific 
behaviors on the form would be redundant, unless the Instructor de- 
sired highly specific feedback on those particular behaviors. 

Furthermore, this study raises the question of how many distinct 
factors exist in such rating instruments. Due to the high intercorrela- 
tlon between pairs of general concept Items in this study, it would seem 
that there are only two or three distinct factors. However, the results 
are not entirely clear as to how many factors there are, thus war- 
ranting further research In this area. 

Finally there is a need for experimental studies to utilize the 
Information of this and similar studies. Using the prescriptive feed- 
back, as the experimental condition, similar to those used by Centra 
(Note 2) and McKeachle (1975), one could assess the true value of 
these ratings and their results to the teachers being evaluated. In this 
way It may be determined whether specific, diagnostic feedback to a 
teacher concerning his teaching performance, as rated by the students, 
actually influences the teacher in a way to change his behavior. 
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p 


E 


D 


0 


PA 


.30 


.48 


.40 


.53 


.33 


.24 


.30 


.24 


.31 


11 


.37 


.50 


.35 


.61 


.44 


.27 


.49 


.21 


.30 


.32 



Table 1 

Correlations Between Global Evaluations 
and General Characteristic Evaluations 

Fo.-m A General characteristics: P E 

Rate the instructor 

Rate ine course 
Form B 

Rate the instructor 

Rate the course 

General characteristics: (with Item used to measure the concept) 

P = Presentation; "The main points of the lecture were clearly 

understood." 

E = Enthusiasm; "The instructor presented the material with 

enthusiasm." 

D = Discussion; "The instructor initiated fruitful and relevant 

discussions." 

0 = Organization; "The Instructor presented the material In a 

well-organized fashion." 

PA = Personal Attention, "The instructor showed consideration and 

empathy for the students." 
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Table 2 

Low Inference Items with Significant Partial Correlations 
with High Inference Variables (p< .01) 

Partial 

Presentation Correlation 

The Instructor pointed out what is important 
to learn In each class session. 

The Instructor summarized the material pre-^ented (.38) 
In each class. 

The instructor defines the students' responsibilities in (.36) 
the course. 

The Instructor used periodic reviews when making (.35) 
logical transistions. 

The Instructor developed eye contact with the students. (.30) 

The Instructor provided alternative ways of learning (.27) 
the course material. 

The Instructor spent time In material relevant to the (.26) 
course objectives. 

Concern was shown for Individual differences. (-25) 

The Instructor was able to answer questions clearly (.25) 
and concisely. 

The Instructor provided practice comprehending (.25) 
course material. 



Organization ; 

The Instructor was able to answer questions clearly (.46) 
ana concisely. 

The Instructor was adequately prepared for each (.46) 
class . 

The content was sequenced In logical fashion. (.45) 

The Instructor spent time In material relevant to (.39) 
the course objectives. 

The Instructor pointed out what was important to (.38) 
learn In each class session. 
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Table 2 (continued} 



The Instructor defines the students' responsibilities tn 
tile course. 

Tiie instructor used different methods and materials. 
The Instructor was confident In his presentations. 
The instructor developed eye contact with the students, 
particular techniques or styles. 

Personal Attention : 

Concern was shown for Individual differences. 

The Instructor provided appropriate material for 
differing rates of progress. 

The Instructor developed eye contact with the students. 

The instructor spent time In material relevant to the 
course objectives. 

The Instructor provided practice comprehending 
course material. 

The Instructor was a dynamic and energetic person. 

The Instructor checked frequently on students' under- 
standing of the material. 

The instructor defines the students' responsibilities 
in the course. 

Enthusaism : 

The Instructor was a dynamic and energetic person. 

The content was sequenced in logical fashion. 

The instructor nsed different methods and materials. 

The Instructor used student responses/contributions 
in developing the lesson. 

The Instructor was adequately prepared for each class. 
The instructor praised the work of the students. 
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Table 2 (continued) 

The Instructor spoke In a monotone, rarely showing (-.30) 
expression in his voice. 

Concern was shown for Individual differences. (.29) 

The Instructor was clear and concise in presentation (.29) 
and explanation of the material. 

The instructor explained the underlying rationale for (.26) 
particular techniques or styles. 

The readings were relevant for the objectives of the (.26) 
course. 



Discussion : 

The Instructor used different methods and materials. (.49) 

The instructor used student responses/contributions (.39) 
in developing the lesson. 

The instructor was a dynamic and energetic person. (.37) 

The Instructor made an effort to show the Interesting (.37) 
nature of the topics. 

The instructor used a variety of teaching methods. (.33) 

The instructor provided alternative ways of learning (.30) 
the course material. 

The instructor used teacher-made materials. (.29) 

The material was too superficial to adequately develop (-.27) 
my skills or concepts. 

The Instructor used periodic reviews when making (.27) 
logical transitions. 

The Instructor developed eye contact with the students. (.26) 

The Instructor used gestures while teaching. (.26) 



Table 3 



Partial 


Correlations Between 


General 


Concepts 




Form A 


P 


E 


D 


0 PA 


presentation 


1.0 








enthusiasm 


.03 


1.0 






discussion 


.08 


.28** 


1.0 




organization 


.31** 


.07 


.14 


1.0 


personal attention 


.19 


.11 


.07 


.10 1.0 


Form B 










presentation 


1.0 








enthusiasm 


.04 


1.0 






discussion 


.05 


.29** 


1.0 




organization 


.25** 


.23** 


.07 


1 .0 


personal attention 


.2^* 


.28** 


.04 


.2H* 1.0 



(* p < .05; ** p < .01) 
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